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ABSTRACT 

Between 1990 and 1993 the Department of Energy's 
precollege mathematics and science program budget grew approximately 
1,250 percent. This report responds to questions about how 
effectively the Department of Energy manages its precollege 
mathematics and science program. The following questions are 
addressed: (1) the appropriateness of the Department of Energy's 
precollege program implementation priorities, (2) the role of project 
evaluations in ensuring rational budget decisions, and (3) whether 
its precollege program helps achieve National Education Goal 5 which 
states that by the year 2000, U.S. students will be first in the 
world in mathematics and science achievement. The report revealed 
that although the Department of Energy invested more than $50 million 
in precollege education in fiscal years 1990-93, it did not 
effectively oversee or direct the program. In addition, the 
Department of Energy did not link budget decisions to project 
evaluation results. Therefore, almost half of its projects have not 
been evaluated. Finally the review revealed that it is doubtful that 
the Department of Energy's precollege program will help achieve 
National Education Goal 5. (2WH) 
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United States 
General Accounting Office 
Washington, D.C. 20548 



Health, Education, and 
Human Services Division 

B-251474 

September 13, 1994 

The Honorable John Glenn 
Chairman, Committee on 

Governmental Affairs 
United Sta s Senate 

Dear Mr. Chairman: 

Our nation's ability to remain the economic world leader depends on its 
citizens' strong mathematics and science skills. Understandably, public 
policymakers and industrial leaders have expressed grave concern about 
precollege students in other industrialized countries significantly 
outperforming American students on recent international mathematics 
and science tests. In response to educational reform and competency 
concerns, former President Bush and the nation's governors developed six 
National Education Goals to be achieved by the year 2000. 1 

In recognition of the Department of Energy's world-class scientists, 
engineers, and technicians, as well as its state-of-the-art laboratories and 
research facilities, the Congress made mathematics and science education 
a nuyor mission for the Department in fiscal year 1991. Consequently, 
Energy's precollege mathematics and science program budget has grown 
approximately 1,250 percent-from approximately $2 million in fiscal year 
1990 to approximately $27 million in fiscal year 1993 (see app. I). 

This report responds to your questions about how effectively Energy 
manages its precollege mathematics and science program. On the basis of 
discussions with your office, we agreed to determine (1) the 
appropriateness of Energy's precollege program implementation priorities, 
(2) the role of project evaluations in ensuring rational budget decisions, 
and (3) whether its precollege program helps achieve National Education 
Goal 5— "By the year 2000, U.S. students will be first in the world in 
mathematics and science achievement." 



Background 0ver ^ e last 10 years > nume «>us reports have charged that many U.S. 

students complete high school scientifically and technologically illiterate. 



In 1904, the Goals 2000: Educate America Act expanded the National Education Goals from six to 
eight The goals address (1) school readiness; (2) school completion; (3) student achievement and 
citizenship; (4) teacher education and professional development; (6) mathematics and science 
achievement^) adult literacy, (7) safe, disciplined, and alcohol- and drug-free schools: and 
(8) parental participation. The National Education Goals were originally developed in 1089. 
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According to these reports, not only are U.S. students less well educated 
than their predecessors, they are also less well trained in mathematics and 
science than their peers in other industrialized countries. Reported 
decreasing student enrollments in science courses, declining achievement 
test scores, and the continuing decline in the number of high-quality 
mathematics and science teachers have highlighted problems in precollege 
mathematics and science instruction. 2 

To remedy this perceived crisis in education, tho Congress conferred 
mathematics and science education responsibilities on the Department of 
Energy and 13 other federal agencies— in addition to the Department of 
Education and the National Science Foundation (nSf). To improve 
mathematics and science education, the Federal Coordinating Council for 
Science, Engineering, and Technology (fccset) 3 Committee on Education 
and Human Resources (cehr) was charged with coordinating the efforts of 
these 16 agencies, fccset developed broad implementation priorities for 
precollege education, including (1) standards for curriculum, teaching, and 
assessment; (2) curriculum, course, and instructional materials; 
(3) systemic reform; and (4) teacher preparation and enhancement These 
priorities were formalized in January 1993 as part of a 5-year strategic 
plan. 



Energy's Precollege 
Mathematics and Science 
Program 



The Department of Energy Science Education Enhancement Act 
authorized Energy to undertake a wide range of precollege education and 
training activities. Some of Energy's activities include making loans of 
equipment and staff to schools; allowing Energy employees to provide 
education-oriented community services; and participating in joint 
programs with schools, businesses, museums, and other community 
partners. Energy manages its precollege program with a decentralized 
organizational structure: Energy's nine national laboratories and 22 of its 
research facilities were given the flexibility to design and implement 
projects using the broad implementation priorities provided by fccset. In 
designing projects, each Energy facility considered fccset's broad 
implementation priorities, as well as its individual areas of specialization 
and local needs. Although the projects vary, most emphasize hands-on 
experiences and fall within three implementation categories: 



^Science, Engineering, and Mathematics Education , Congressional Research Service Issue Brief, 
(WashingtonTaC.: 1§92), p. 1. 

*tn November 1993, President Clinton established by executive order a cabinet level National Science 
and Technology Council (NSTC) to coordinate science, space, and technology policies throughout the 
federal government The establishment of NSTC consolidated the responsibilities previously carried 
out by a number of agencies, including FCCSET. 
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• Teacher enhancement— These projects attempt to further the content 
knowledge, skills, and experiences of teachers already in the workforce. 
Teacher enhancement projects typically provide teachers with 
opportunities to (1) work on a variety of scientific and technical subjects 
as a member of an Energy laboratory research team, (2) train with 
mentors who assist them with in-school science experiments, or (3) obtain 
sophisticated computer training. 

• Student supports — These projects seek to reward students for outstanding 
achievement, afford them enrichment experiences, and furnish them with 
supplementary educational services such as tutoring and mentoring. 
Student support projects typically allow students opportunities to 

(1) participate in cutting-edge research at an Energy laboratory, (2) attend 
scientific lectures and demonstrations, and (3) study emerging topics such 
as environmental energy. 

• Systemic reform— These projects aim to improve education by changing 
all aspects of an educational system. Systemic reform projects typically 
involve key education stakeholders— students, teachers, administrators, 
policymakers, and parents— in collaborative efforts to (1) create goals and 
standards for ail students, (2) develop related curricula and instructional 
materials, and (3) provide professional development for teachers. 



ReSllltS in Brief Although Energy invested more than $50 million in precoUege education in 

fiscal years 1990-93, it did not effectively oversee or direct the program. 
For example, although research findings indicate that systemic reform 
may have the greatest potential to improve student learning, projects in 
this implementation category constituted the smallest share of Energy's 
precoUege budget, about 11 percent In contrast, Energy used about 
70 percent of its precoUege budget to finance teacher enhancement 
projects, even though research suggests that these projects may be 
ineffective at increasing student achievement. 

To compound problems, Energy did not link budget decisions to project 
evaluation results. As a result, Energy had not evaluated almost half of its 
17 most resource-intensive projects at the time of our review; for those 
projects with evaluation reports, aU were inadequate. Nonetheless, Energy 
substantial increased funding for most of these projects— in one case by 
over 1,700 percent 

In addition, it is doubtful whether Energy's precoUege program wiU help 
achieve National Education Goal 6. In this regard, Energy's projects 
typically do not focus on student achievement, which is central to 
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achieving this goal. In fact, more than half of Energy's most 
resource-intensive projects did not directly include improving student 
achievement as an objective. 

During our review, Energy indicated recognition of the need to pay closer 
attention to managing its precollege program. To correct this situation, 
Energy recently drafted an agency-specific strategic plan. In addition, 
officials said they plan to restructure their program to ensure that all 
projects are evaluated and linked more clearly to National Education Goal 
5. 



^ Our review of Energy's precollege mathematics and science program 
bCOpe ana focused primarily on its 288 fiscal year 1992 projects. To accomplish our 

Methodology objectives, we conducted site visits at Energy's headquarters in 

Washington, D.C., and eight laboratories and research facilities in 
California, Illinois, New Mexico, Tennessee, and Utah, which administered 
the most resource-intensive precollege projects. 4 During these site visits, 
we interviewed laboratory personnel as well as school administrators, 
teachers, and students involved in the precollege program. 

To obtain an overview of the program, we collected general information 
on Energy's entire array of fiscal year 1992 precollege mathematics and 
science projects and reviewed the literature on the relationship between 
teacher quality and subsequent student achievement. For the 17 most 
resource-intensive projects, we collected and analyzed budget data for 
fiscal years 1990-93 (see app. HI) and, when available, evaluation reports. 
Although constituting less than 6 percent of Energy's program portfolio, 
these 17 projects accounted for 50 percent ($11 million) of total program 
dollars in fiscal year 1992. To determine whether the evaluation reports 
provided reliable information on project effectiveness, we also conducted 
a technical review of the nine project evaluation reports Energy officials 
submitted to our staff (see app. IV). 

We conducted our work between November 1992 and July 1994 in 
accordance with generally accepted government auditing standards. 



♦To identify the most resource-intensive projects, we examined the extent to which both financial and 
personnel resources were used We selected 16 projects with the largest budgets and 1 project that 
involved considerably more Energy personnel than other projects. 
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Focus on Teacher 
Enhancement 
Projects Is 
Questionable 



Energy's decision to concentrate its precoP 0<, e program resources on 
teacher enhancement projects, which account for more than two-thirds of 
the program budget, is a questionable implementation strategy. In an 
earlier report, we found no evidence that training programs to upgrade 
existing science and mathematics teachers' skills will improve teaching 
effectiveness. 6 In reviewing more recent studies, we found mixed results: 
in some instances, researchers found small yet statistically significant 
positive correlations between teacher knowledge and student 
achievement; in others, researchers failed to demonstrate any significant 
correlations. Conversely, current literature suggests that systemic reform 
measures, such as high-quality curriculum development, may hold the 
most promise for improving academic achievement and realising the 
national math and science goals. 



Precollege Program Both in budget dollars and project numbers, Energy devoted most of its 

Heavily Weighted Toward precollege program resources to teacher enhancement projects during 
Teacher Enhancement fiscal year 1992. Regarding the program budget, Energy spent aboux 70 

Proiects percent ($15.4 million) of all precollege program dollars to upgrade 

J teachers' mathematics and science skills — 45 percent exclusively for 

teacher enhancement and another 25 percent for teacher enhancement 
combined with a student support component For the residual, about 19 
percent ($4.2 million) of Energy's precollege budget focused exclusively 
on student support; systemic reform efforts constituted just 11 percent 
($2.4 million). Regarding the total number of projects, 157 of 288 
(55 percent) had a substantial teacher enhancemer' omponent, another 
113 projects (39 percent) focused exclusively on student support, 14 
projects (5 percent) involved systemic reform efforts, and 4 projects 
(1 percent) included activities that did not specifically involve teachers 
and students (see table 1). 



6 New Directions for Federal Programs to Aid Mathe matics and Science Teaching (GA0/PEMD-84-6, 
Mar. 6, 1984). 31 

r n SfV> Pfttfe * 7 GAO/EEHS-94-208 Precollege Math and Science Education 



B-251474 



Table 1: Precollege Program Heavily 
Weighted Toward Teacher 
Enhancement Projects 



Fiscal year 1992 



Implementation priorities 


Budget (in 
thousands) 


Percent 


Number 


Percent 


Teacher enhancement 


$9,91 1 


45 


100 


35 


Teacher enhancement and student 
support 


1 5,524 


25 


57 


20 


Student support 


4,187 


19 


113 


39 


Systemic reform 


2,448 


11 


14 


5 


Other* 


95 


b 


4 


1 


Total 


$22,165 


100 


288 


100 



•"Other" includes activities thai do not specifically Involve students and teachers, such as efforts 
by Energy's staff to develop a catalog listing precollege physics projects. 



b Lessthan 1 percent. 



No Strong Relationship 
Between Teacher 
Enhancement and Student 
Achievement 



Research has failed to show conclusively a relationship between teacher 
enhancement and student achievement 6 We based our finding on the 
results of a 1984 gao report and a review of recent studies that examined 
the effect of teacher quality on student achievement (see bibliography, p. 
25). In our earlier report, we found no evidence that training programs to 
upgrade mathematics and science teachers' skills improved student 
achievement 7 That analysis was based first on an nsf study that compared 
the achievement of eighth and eleventh grade students whose teachers 
participated in nsf institutes to those who did not The study showed that 
teacher participation in nsf institutes had a positive effect on eleventh 
grade students' science and mathematics achievement However, institute 
participation did not have a statistically significant effect on eighth grade 
student achievement in either science or mathematics. Second, several 
general studies from the 1960s and 1970s as a group failed to show a 
consistent relationship between teacher knowledge and student 
achievement 



More recent studies have also failed to demonstrate a strong relationship 
between teacher enhancement/knowledge and student achievement Two 
of the most prominent studies conducted between 1984 and 1994 that 
examined this relationship reported mixed results: 



^Although research in this area has been limited, the studies we identified continue to be cited in 
current research, and the findings remain unchallenged. 

7 New Directions for Federal Programs to Aid Mathematics and Science Teaching . 
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• A 1992 nsf study, which analyzed teacher transcript and student test data 
from the National Education Longitudinal Study of 1988 (NEL&88), 8 showed 
a statistically significant relationship between eighth grade students 9 
mathematics achievement and their teachers' preparation. 0 Specifically, 
students whose teachers had mayored in mathematics performed slightly 
better than those whose teachers had mgjored in education only. 
However, no statistically significant relationship existed for science. 

• A 1994 Chicago Academy of Sciences study, which analyzed the effect of 
teachers' participation in 4-week summer science workshops, reported a 
small, but statistically significant, increase in the level of science 
achievement for seventh grade students of participants. 10 For these 
students, the average science achievement score increased from 47.1 to 
49.6. However, no statistically significant change occurred for either sixth 
or eighth grade science students. 

At least four explanations for the weak relationship between teachers' 
knowledge or participation in training programs and student achievement 
exist First, most training programs generally involve one-time, relatively 
short events with little or no follow-up. For example, a single 4- to 8-week 
summer research experience probably would not produce a dramatic 
change in a teacher's effectiveness. Second, teacher training programs are 
subject to self-selection bias; that is, in-service training often attracts 
exemplary or highly motivated teachers. Exposing such teachers to 
short-term workshop training may not significantly add to their teaching 
effectiveness. Third, the most knowledgeable teachers may not be the best 
teachers. Other factors besides teacher knowledge — such as enthusiasm, 
confidence, and organization of class time — may determine student 
achievement. Fourth, some researchers suggest that current student 
assessments inadequately measure the higher order, problem-solving 
skills, which could be affected by teacher training and knowledge. 



*NELS:88 Is a nationally representative sample of 26,436 eighth-grade students clustered within 1,052 
schools. 

•Senta Raizen and Theodore Britton, "Science and Mathematics Teachers* Indicators of Science and 
Mathematics Education in 1992, National Science Foundation (Washington, D.C.: 1993), pp. 85-113. 

I0 Jon D. Miller, Enriching Middle School Science: A Final Evaluation of the 1991-92 Columbia College 
Workshops Utilizing an Innovative Approach to the Teaching of Science. Chicago Academy of Sciences 
(NSF grant TPE 89«128), (Chicago: 1994), pp. 30-31. 
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Systemic Reform 
Considered Promising 
Approach for Increasing 
Student Achievement 



Energy has implemented few systemic reform projects, even though 
current educational literature suggests systemic reform may have the 
greatest potential for improving student learning. 11 Systemic reform is 
promising because it (1) attempts to stimulate change in many or all 
components of the educational system simultaneously; (2) establishes 
clear standards for what students should know and be able to do; and 
(3) involves key educational stakeholders— students, teachers, 
administrators, teacher educators, textbook publishers, policymakers, and 
parents — at all levels of the education system— national, state, district, and 
school. 



Under systemic reform, teacher training ideally takes place in an 
environment that supports new curricula or teaching techniques learned 
during training. However, many educators believe that retraining 
individual teachers will have little measurable impact on student 
achievement if the education system is not prepared to absorb 
improvements. For example, although teacher enhancement programs 
may provide teachers the knowledge, skills, and enthusiasm essential to 
implement new curricula, teachers can rarely implement and sustain a 
new program if their school support systems and attitudes of 
administrators, colleagues, and parents have not changed. 



Additional Program 
Diversity Could Reduce 
Program Risk 



Given the evidence cited, Energy's heavy investment in teacher 
enhancement projects substantially increases its risk of not improving 
student achievement in mathematics and science. However, Energy could 
reduce this risk by changing its mix of projects to balance the program, 
much like financial advisers do by diversifying investment portfolios. In 
managing uncertainty, financial advisers minimize the risk of loss by 
acquiring a variety of investment vehicles; thus, good returns from one 
investment counterbalance poor returns from another. Building on this 
analogy, Energy could view its precollege program as a collection of 
investment vehicles assembled to meet an investment goal — improved 
student achievement in mathematics and science. Thus, given the 
uncertainty of its projects' educational payoffs, a more diverse program 
portfolio would enhance the likelihood of Energy's achieving its program 
goal. 

To address these concerns, Energy developed a strategic plan, which 
identifies agency-specific precollege goals and objectives in March 1994. 



u Much of this literature is cited in Marshall Smith and Jennifer O'Day, •Systemic School Reform," 
Politics of Education Assodatior Yearbook , (1990), pp. 233-267. 



Page 8 



j 0 GACWHEHS-94-208 PrecoUege Math and Science Education 



B-251474 



Energy envisions using the strategic plan, which should be in place by 
November 1994, to help it create a program strategy that supports the best 
mix of projects and minimizes risk. In addition, Ener,# plans to look at the 
feasibility of eliminating projects that do not support its strategic plan and 
restructuring all teacher enhancement projects to include systemic reform 
elements, such as follow-up support beginning in fiscal year 1996. 



Energy did not link budget decisions to project evaluations. Until recently, 
Energy neither required project evaluations nor ensured their adequacy 
when its research facilities conducted them. For example, Energy elected 
not to evaluate eight of its 17 mdst resource-intensive projects. When 
conducted, evaluations were of poor quality. For instance, all projects with 
evaluations contained discrediting technical flaws — such as insufficient 
sample sizes, the absence of statistical tests, and insufficient supporting 
data— that potentially invalidated ciny evaluation findings (see app. IV). 

Energy's limited use of program evaluation reflected its management 
priorities. According to Energy officials, the Department did not 
emphasize effectiveness evaluations because program expansion was its 
primary objective. These officials also said that Energy lacked th^ 
capacity— staff, funds, and expertise — to design and monitor evaluations 
for such a vast array of projects. Consequently, in the absence of sufficient 
evaluation results, Energy substantially increased project budgets on the 
basis of self-reported data, such as customer satisfaction surveys, or 
anecdotal data such as participant testimonials, requests to participate h; 
particular projects, or other popularity indicators. Consider the following 
examples: 



Budget Decisions Not 
Linked to Project 
Evaluations 
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Energy increased the budgets for four projects with no evaluations, with 
increases rau ging from 62 to 912 percent from fiscal year 1991 to 1992 (see 
fig. I). 12 




| 1 FY 1991 
H FY 1992 



Notes: 



■The Science/Math Carnival (SNL-L) project budget was Increased from approximately $3,000 to 
$26,000 or by 912 percent. 

'The Bay Area Science and Technology Education Collaboration (LBL) project budget was 
increased from approximately $288,000 to $466,000 or by 62 percent. 

c The Environmental Management Precollege Analytical Chemistry (AWU) project budget was 
increased from approximately $225,000 to $61 5,000 or by 1 73 percent. 

<*The National Science Explorers (ANL) project budget was Increased from approximately 
$318,000 to $693,000 or by 118 percent, 



"Projects shown in figure 1 began in fiscal year 1991. 

12 
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• Energy increased the budgets for five projects with inadequate 
evaluations, with increases ranging from 113 to 1,731 percent from fiscal 
year 1990 to 1992 (see fig. 2). 



Figure 2: Project Budgets Increased ^^^^^M^B 
Substantially Despite Inadequate ooftara in thousand* 

Evaluations 2soo 



/ 




A(*113%y B(+372V 0(^1731%)- D (+;»•%)• E(+350%)* 

ProJtcVbudflrt tncrMMS FY 1090-92 



] FY 1990 




FY 1991 
FY 1992 



Notes: 



•The Environmental Education Outreach for Minorities (BNL) project budget was increased from 
approximately $93,000 to $199,000 or by 1 13 percent. 

b The Teacher Research Associates (ORiSE) project budget was increased from approximately 
$49,000 to $231 ,000 or by 372 percent. 

°The OPTIONS (PNL) project budget was increased from approximately $50,000 to $91 5,000 or 
by 1 ,731 percent, 

The Teacher Research Associates (AWU) project budget was increased from approximately 
$526,000 to $1 ..779,000 or by 238 percent. 

•The Science Advisors (SNL-A) project budget was Increased from approximately $545,000 to 
$2,500,000 or by 359 percent. 



13 
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On the basis of our findings, Energy needs to change the way it views the 
relationship between project implementation and evaluation. Energy 
should view program evaluation as an integral part of program 
management; rather than perceiving the two as mutually exclusive. 
Because quality program evaluation is expensive, Energy must accept the 
unavoidable trade-off that it must fund fewer projects with stronger 
evaluation components. 

To address its program evaluation shortcomings, Energy began a 
partnership with the National Center for Improving Science Education to 
jointly develop a system for ongoing evaluation of its precollege program 
in May 1992. In addition, Energy established an evaluation guidance 
committee responsible for developing a long-range implementation plan 
for project evaluation. Energy officials also said that beginning in fiscal 
year 1995, the Department will require each precollege project to have an 
evaluation component as a prerequisite for funding. 



Energy has greatly diminished the prospect of its program's helping to 
achieve National Education Goal 5 — making American students first in the 
world in mathematics and science — by not emphasizing student 
achievement Although it is the essence of Goal 5, increasing student 
achievement is the key objective in only 7 of the 17 most 
resource-intensive program projects. The remaining 10 projects generally 
seek to improve students' attitudes toward mathematics and science and 
motivate them to eventually pursue science careers by improving their 
perceptions of scientists. 

Moreover, Energy's evaluation process did not focus on student 
achievement: only one of the nine projects that were evaluated tried to 
measure student achievement. In fact, Energy's projects are seldom clearly 
linked to National Education Goal 5. For example, two of seven program 
managers we interviewed were unfamiliar with the National Education 
Goals. In addition, none of the program managers could demonstrate how 
their projects helped improve student achievement. Generally, these 
managers told us their projects aim to increase mathematics and science 
literacy and promote science as a career, not improve student 
achievement 



In the early 1990s, Energy did not effectively manage its precollege 
mathematics and science program. First, the Department jeopardized the 
program's success by not using a risk management strategy to administer 
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the program's projects. Second, Energy forfeited an invaluable 
management tool by taking a lax approach to program evaluation. Third, 
Energy greatly reduced its probability of helping achieve National 
Education Goal 5 by implementing a variety of projects that did not clearly 
seek to improve student achievement. 

In response to concerns raised during our review, Energy officials 
announced plans to undertake several initiatives to substantially improve 
this program's management and evaluation functions. These initiatives 
constitute an important step toward effective program management. 
However, the depth of executive support for these initiatives and their 
subsequent staying power were uncertain at the time our review was 
completed. If ongoing changes in Energy's management philosophy are 
fully implemented, needed program improvements could ultimately result. 



In continuing the Department's efforts to improve management of the 
precollege mathematics and science education program, we recommend 
that the Secretary cf Energy strengthen its management role. Specifically, 
the Secretary should 

• place greater emphasis on balancing the program by increasing the 
proportion of systemic reform projects; 

• strengthen its evaluation component so that it serves as a basis for 
(1) improving projects; (2) making informed budget decisions about 
terminating, retaining, and expanding projects; and (3) measuring gains in 
student achievement; and 

• restructure or discontinue all projects that do not clearly support National 
Education Goal 5 — increasing students' mathematics and science 
achievement. 



Recommendations to 
the Secretary of 
Energy 
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Department of Energy officials who reviewed a draft of this report 
generally agreed with our findings. As requested, unless you publicly 
announce its contents earlier, we plan no further distribution of this report 
until 30 days from its issue date. At that time, we will send copies of this 
report to the appropriate House and Senate Committees, the Secretary of 
Energy, and other interested parties. If you or your staff have any 
questions about this report, please call me on (202) 512-7014 or Cornelia 
Blanchette, Associate Director, on (202) 512-8403. The imgor contributors 
to this report are listed in appendix V. 

Sincerely yours, 




Linda G. Morra 
Director, Education and 
Employment Issues 
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AWU 


Associated Western Universities 


BNL 


Brookhaven National Laboratory 


FCCSET 


Federal Coordinating Council for Science, Engineering, and 




Technology 


CEKR 


Committee on Education and Kuman Resources 


LANL 


Los Alamos National Laboratory 


LBL 


Lawrence Berkeley Laboratory 


IXNL 


Lawrence Iivermore National Laboratory 


NSF 


National Science Foundation 


ORISE 


Oak Ridge Institute for Science and Education 


ORNL 


Oak Ridge National Laboratory 


PNL 


Pacific Northwest Laboratory 


SNL-A 


Sandia National Laboratory-Albuquerque 


SNL-L 


Sandia National Laboratory-Iivermore 
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Appendix I 

Precollege Program Budget Increased 
Significantly (Fiscal Years 1990-93) 



40 Dollars In millions 

35 

30 

27 

25 



22 
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Note: Actual budget totals shown for fiscal yaars 1990-92. Appropriated budget totals shown for 
fiscal year 1993. 
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Appendix II 



Overview of 17 Most Resource-Intensive 
Projects (Fiscal Year 1992) 





Energy 
facility 


Project name 


Project objective 


Fiscal year Implementation 
1992 budget priority 


Projects with no evaluation (eight projects) 


Ames Laboratory 
(Iowa) 


Adventures in 
Supercomputing 


Introduce teachers to the use of 
high-performance computers in 
mathematics and science 
instruction. 


Teacher enhancement 

$315,888 


Argonne National 
Laboratory (Illinois) 


National Science Explorers 


Provide teachers with science 
videos, instructional guides, and 
training workshops to improve 
science teaching. 


Teacher enhancement 

693,000 


Associated Western 
Universities (Utah) 


Environmental Management 
Precollege Analytical Chemistry 


Provide students with an 
opportunity to take a 
college-level analytical chemistry 
course. 


Teacher 

enhancement and 
student support 

615,014 



Provide hands-on activities, Systemic reform 

curriculum development, 

instructional materials, and 

training districtwide to enhance 

classroom teaching and learning 

of science, mathematics, and 

technology. 466,000 

Provide teachers and students Systemic reform 

with research opportunities, 

instructional materials, and 

technical support to increase the 

effectiveness of mathematics 

and science education 

districtwide. 244,000 



Oak Ridge National 
Laboratory (Tennessee) 


Adventures in Supercomputing 


Provide teachers and students 
with access to and training on 
high-performance computers to 
improve mathematics and 
science instruction. 


398,000 


Teacher 

enhancement and 
student support 


Oak Ridge National 
Laboratory (Tennessee) 


Preparation & Enhancement 


Provide teachers with summer 
research opportunities and 
training to improve mathematics 
and science instruction. 


260,000 


Teacher enhancement 


Sandia Nations' 

Laboratory-Liverrnore 

(California) 


Science/Math Carnival 


Provide scientific demonstrations 
to improve mathematics and 
science teaching. 


26,306 


Teacher enhancement 


Subtotal 






$3,018,208 
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Lawrence Berkeley Bay Area Science & Technology 
Laboratory (California) Education Collaboration 



Oak Ridge institute for Science & Mathematics Action 
Science and Education for Revitalized Teaching 
(Tennessee) 
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Appendix II 

Overview of 17 Moit Reaource-Inteiwive 
Project* (Fiscal Year 1992) 



Energy 
facility 


Project name 


Project objective 


Flecal year Implementation 
1992 budget priority 


Projecte with evaluation (nine projacta) 


Argonne National 
Laboratory (Illinois) 


Chicago Science Explorers 


Provide teachers and students 
witn nanos-on acuviiies, neia 
trips, and videos to improve 
science instruction and learning. 


Teacher 

OllliCMIv>OHIt?(ll CM IU 

student support 

$531,000 


Associated Western 
Universities (Utah) 


Teacher Research Associates 


rrovioe summer researcn 
opportunities to teachers. 


1 Oovl Id Ol II ICM IwOI 1 ICl 11 

1,779,269 


Brookhaven National 
Laboratory (New York) 


Environmental Education 
Outreach for Minorities 


Provide students with an 
opportunity to take a 
college-level environmental 
science course. 


Student support » 

199,211 


Brookhaven National 
Laboratory (New York) 


Northeast Consortium for 
Minorities 


Provide students with an 
opportunity to take college-level 
science courses. 


Student support 

229,381 


Los Alamos National 
Laboratory (New 
Mexico) 


Students Watching Over Our 
Planet Earth 


Provide materials, teacher 
training, and student instruction 
on environmental concerns. 


Teacher 

enhancement and 
1,016,900 student support 


National Laboratory 
(California) 


National Education 

Supercomputer 

Program 


Provide access to and training 
on high-performance computers 
to students and teachers. 


Teacher 

enhancement and 
726,000 student support 


Oak Ridge Institute for 
Science and Education 
(Tennessee) 


Teacher Research Associates 


Provide summer research 
opportunities for teachers. 


I eacner ennancerneni 

230,600 


Pacific Northwest 

Laboratory 

(Washington) 


OPTIONS in Science 


Provide students with 
high-quality mathematics and 
science education by enhancing 
teachers 1 instructional strategies 
and ability to develop curriculum 
through a statewide systemic 
reform effort. 


Systemic reform 

915,370 


Sandia National 
Laboratory- 
Albuquerque (New 
Mexico) 


Science Advisors 


Provide a scientist in the school 
who offers technical assistance 
to teachers and students and 
participates in activities to 
support science (i.e., science 
fairs). 


Teacher 

enhancement and 
student support 

2,500,000 


Subtotal 






$8,127,731 


Total 






$11,145,939 
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Appendix HI 

Percent Change in Budget for Energy's 
17 Most Resource-Intensive Projects 
(Fiscal Years 1990-93) 









Fiscal 
year 


Fiscal year 1991 


Fiscal year 1992 


Fiscal year 1993 






1990 




Percent 




Percent 




Percent 


Energy facility 


Project name 


Budget 


Budget 


change 


Budget 


change 


Budget 


change 


Projects with no evaluation (eight projects) 


Ames Laboratory (Iowa) 


Adventures in 
Supercomputing 








$315,888 


N/A 


$650,000 


106 


Argonne National 
Laboratory (Illinois) 


National Science 
Explorers 




$318,000 


N/A 


693,000 


118 


507,000 


-27 


Associated Western 
Universities (Utah) 


Environmental 
Management 
Precollege 
Analytical Chemistry 




225,117 


N/A 


615,014 


173 


601,870 


-2 


Lawrence Berkeley 
Laboratory (California) 


Bay Area Science 
& Technology 
Education 
Collaboration 




288,300 


N/A 


466,000 


62 


515,000 


11 


Oak Ridge Institute for 
Science and Education 
(Tennessee) 


Science & 
Mathematics Action 
for Revitalized 
Teaching 


24,100 


279,100 


1,058 


244,000 


-13 


260,000 


7 


Oak Ridge National 
Laboratory (Tennessee) 


Adventures in 
Supercomputing 








398,000 


N/A 


920,000 


131 


Oak Ridge National 
Laboratory (Tennessee) 


Teacher 
Preparation & 
Enhancement 








260,000 


N/A 


260,000 


N/A 


Sandia National 
Laboratory - Livermore 
(California) 


Science/Math 
Carnival 




2,600 


N/A 


26,306 


912 


29,000 


10 


Subtotal 




$24,100 


$1,113,117 


4,519 


$3,018,208 


171 


$3,742,870 


24 
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Appendix III 

Percent Change In Budget for Energy's 
17 Moit Reftonrce-Inteni&ve Projects 
(Fiscal Years 1990-93) 







Fiscal 
year 


Fiscal year 1991 


Fiscal year 1992 


Fiscal year 1993 


Energy facility 


Project name 


1990 
Budget 


Budget 


Percent 
change 


Budget 


Percent 
change 


Budget 


Percent 
change 


Projects with evaluation (nine projects) 


Argonne National 
Laboratory (Chicago) 


Chicago Science 
Explorers 


$613,000 


$570,000 


-7 


$531,000 


-7 


$750,000 


41 


Associated Western 
Universities (Utah) 


Teacher Research 
Associates 


525,791 


1,105,519 


110 


1,779,269 


61 


1,731,326 


- 3 


Brookhaven National 
Laboratory (New York) 


Environmental 
Education 
Outreach for 
Minorities 


93,351 


125,358 


34 


199,211 


59 


168,500 


- 15 


Brookhaven National 
Laboratory (New York) 


Northeast 
Consortium for 
Minorities 




122,503 


N/A 


229,381 


87 


249,703 


9 


Los Alamos National 
Laboratory (New 
Mexico) 


Students Watching 
Over Our Planet 
Earth 


85,000 


750,000 


782 


1,016,900 


36 


289,000 


-72 


Lawrence Livermore 
National Laboratory 
(California) 


National Education 
Supercomputer 




340,000 


N/A 


726,000 


114 


403,000 


-45 


Oak Ridge Institute for 
Science and Education 
(Tennessee) 


Teacher Research 
Associates 




139,700 


186 


230,600 


65 


193,000 


-16 


Pacific Northwest 

Laboratory 

(Washington) 


OPTIONS in 
Science 


50,000 


629 t 762 


1,160 


915,370 


45 


374,033 


-59 


Sandia National 
Laboratory - 
Albuquerque (New 
Mexico) 


Science Advisors 


545,000 


1,400,000 


157 


2,500,000 


79 


2,200,000 


-12 


Subtotal 




$1,961,042 


$5,182,842 


164 


$8,127,731 


57 


6,358,559 


-22 


Total 




$1,985,142 


$6,295,959 


217 


$11,145,939 


77 


$10,101,429 


•9 



Note: N/A represents not applicable. 
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Appendix IV 



Evaluation Report Results for Nine Projects 
With Evaluations 



Number of 

reported QAO 
Technical limitation! findings conclusion 



Energy facility/project 
name 


Fiscal year 
1992 budget 


No supporting 
data 


No statistical 
tests 


Insufficient 
sample size* 


Strong 


Weak 


Adequate 

report? 


ANL: Chicago Science Explorers 


$ 531,000 


✓ 




✓ 


0 


3 


no 


AWU: Teacher Research 


1,779,269 








0 


5 


no 


BNL: Environmental Education 
Outreach for Minorities 


199,211 


✓ 


✓ 


✓ 


0 


4 


no 


BNL: Northeast Consortium for 
Minorities 


229,381 


✓ 






0 


1 


no 


LANL: Students Watching Over 
Planet Earth 


1,016,900 






✓ 


2 


1 


no 


LLNL: National Education 
Supercomputer Program 


726,000 








0 


1 


no 


ORISE: Teacher Research 
Associates 


230,600 




✓ 




0 


5 


no 


PNL: OPTIONS in Science 


915,370 








0 


2 


no 


SNL-A: Science Advisors 


$2,500,000 








1 


8 


no 



Criteria 



To assess the adequacy of Energy's project evaluations, we reviewed evaluation reports submitted for nine projects. The purpose of our review was to 
determine the extent to which the reports provided accurate and adequate Information about each project. For each report, we Identified findings about 
the effects of the project on (1 ) teaching skills in science and mathematics, (2) teacher comfort with teaching science and mathematics, (3) teacher 
attitudes toward science and mathematics, (4) teacher knowledge of science and mathematics, (5) use of nonK:ture methods, (6) student mathematics 
and science skills, (7) student attitudes toward science and mathematics, and (8) student knowledge of sclem, and mathematics. 

We classified each report finding as strong or weak based on the stren^ of the evaluation methodology. We considered a finding to be strong If 
(1 ) supporting data were presented In the evaluation report and (2) when appropriate, a statistical significance test was done with at least 30 cases 
using a significance level of .05. Strong findings may have contained positive or negative conclusions about a project. We based our classifications 
solely on the evaluation reports. We conslderd an evaluation report to be adequate If it contained only strong findings. 

•While nonparametric tests may be used with sample sizes smaller than 30, we found no evidence that such tests were used. 
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